Conventional database processing techniques typically verify the accuracy of data prior to making an update to the primary database. It is common practice that if during the verification process, any of the updates are found to be in error, the errors are reported and none of the modifications are made. For example, a collective group of updates can include the following: addition of new records to one or more database tables, creation (“add”) of one or more new tables, deletion (“drop”) of one or more existing tables, and change of a value already stored in one or more specific fields within one or more database tables.
For example, a relational database is commonly used as the primary data store for applications and other data consumers. Because the nature of a relational database is to link multiple tables through a system of dependencies, a single update to a single table will often require updates to each of the table's dependent tables. If an error is discovered while attempting to update one in a series of dependent tables, the tables that have already been updated must be restored to their prior state to avoid data instability.
Updates can involve thousands of records on multiple database tables. The shear size and complexity of such a database update creates problems with conventional means of data verification prior to update of the primary database. For example, the database feature known as “roll back” is widely used to return tables to their initial state if an error is discovered during an update that spans multiple tables. However, roll back is not an option with many commercially available databases when the database update becomes sufficiently large. This is because many databases have a feature known as “auto commit” that commits the updates to the database automatically after a set number of updates have been made. Another conventional technique for supporting verification during an update is to store all proposed updates in memory until the verification process is complete. This strategy has the disadvantage of tying up large amounts of system resources. This is especially true for large files containing many updates. The lack of system resources can in turn cause errors in application processing when sufficient memory is not available.
There is a need for a system and method for processing large update files that avoids the problems identified above.